-1- 

METHOD AND APPARATUS USING MULTIPLE SENSORS 
IN A DEVICE WITH A DISPLAY 

REFERENCE TO RELATED APPLICATION 
The present invention claims priority from a 
U.S. Provisional application having serial number 
60/218,748, filed on July 17, 2000 and entitled "METHOD 
AND APPARATUS USING MULTIPLE SENSORS IN A MOBILE 
DEVICE." 

BACKGROUND OF THE INVENTION 

The present invention relates to devices 
with displays. In particular, the present invention 
relates to computing and mobile devices. 

Mobile devices, such as personal 
information managers (PIMs) , cellular telephones, 
pagers, watches, and wearable computers typically 
include one or more buttons or touch screens through 
which the . mobile device receives explicit 
instructions from the user. For example, the user 
can press buttons to explicitly instruct the device 
to enter a full -power mode, activate an application, 
or scroll through an image on the display. 

Although the devices are responsive to 
information provided through such explicit 
instructions, they are generally not responsive to 
information that is present in the manner in which 
the device is being handled by the user. For example, 
the devices do not automatically enter a full -power 
mode, even when the user is holding the device in a 
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manner that is consistent with wanting to use the 
device . 

The reason these devices are not responsive 
to such handling information is that they typically 
5 are not equipped with the sensors needed to detect 
the information nor with the software needed to 
interpret the information. 

Because these devices are generally not 
responsive to the manner in which the user is holding 
10 the device, the user is forced to enter explicit 
instructions into the device to achieve various 
functions. In light of this, mobile devices are 
needed that can sense how they are being handled in 
order to perform certain background functions that 
15 expand the functionality of the mobile device without 
requiring the user to perform any additional actions. 

SUMMARY OF THE INVENTION 
In a device having a display, at least one 
sensor signal is generated from a sensor in the 
2 0 device. One or more context values are then 

generated from the sensor signal. The context values 
indicate how the device is situated relative to one 
or more objects. At least one of the context values 
is then used to control the operation of one or more 
25 aspects of the device. 

The invention includes several aspects. In 
one aspect, an' image on a display is scrolled at a 
rate that is based on the difference between a 
current tilt angle and a tilt angle when tilt 
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scrolling was activated. A further aspect of the 
invention adjusts the contrast of a display based on 
the tilt angle of the display. 

Other aspects of the invention control the 
5 power mode of the device based on whether it is being 
handled, its orientation, and/or whether it is being 
gestured toward. 

Still further aspects of the invention 
activate applications based on the device being in a 
10 particular orientation while being held by the user. 

BRIEF DESCRIPTION OF THE DRAWINGS 

FIG. 1 is a block diagram of the components 
of a mobile device under one embodiment of the 
present invention. 
15 FIG. 2 is a bottom view of a mobile device 

of one embodiment of the present invention. 

FIG. 3 is a front view of the mobile device 

of FIG. 2. 

FIG. 4 is a back view of the mobile device 

20 of FIG. 2. 

FIG. 5 is a left side view of the mobile 
device of FIG. 2. 

FIG. 6 is a right side view of the mobile 
device of FIG. 2. 
25 FIG. 7 is a graph of distance between a 

user and a mobile device as a function of proximity 
sensor levels. 



FIG. 8 is a block diagram of components 
used to practice several embodiments of the present 
invention. 

FIG. 9 is a graph of left-right tilt versus 
5 forward-back tilt showing those tilt angles that 
activate an audio device. 

FIG. 10 is a front view of a "mobile device 
in a portrait orientation. 

FIG. 11 is a front view of a mobile device 
10 in a landscape orientation, 
g FIG. 12 is a chart showing the display 

J orientations for various combinations of forward/back 

Sj tilt and left/right tilt. 

HI DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

S| 15 FIG. 1 is a block diagram of a mobile 

device 200, which is an exemplary environment for 
y embodiments of the present invention. Mobile device 

O 200 includes a microprocessor 202, memory 204, 

input/output (I/O) components 206, and a 
N= 20 communication interface 208 for communicating with 

remote computers or other mobile devices. In one 
embodiment, the afore-mentioned components are 
coupled for communication with one another over a 
suitable bus 210. 
25 Memory 204 is implemented as a non-volatile 

electronic memory such as a random access memory 
(RAM) with a battery back-up module (not shown) such 
that information stored in memory 204 is not lost 
when the general power to mobile device 200 is shut 
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down. A portion of memory 2 04 is preferably allocated 
as addressable memory for program execution, while 
another portion of memory 204 is preferably used for 
storage, such as to simulate storage on a disk drive. 
5 Memory 2 04 includes an operating system 

212, application programs 214, and an object store 
216. During operation, operating system 212 is 
preferably executed by processor 202 from memory 204. 
Operating system 212, in one preferred embodiment, is 

10 a WINDOWS® CE brand operating system commercially 
available from Microsoft Corporation. Operating 
system 212 is preferably designed for mobile devices, 
and implements database features that can be utilized 
by applications 214 through a set of exposed 

15 application programming interfaces and methods. The 
objects in object store 216 are maintained by 
applications 214 and operating system 212 at least 
partially in response to calls to the exposed 
application programming interfaces and methods. 

20 Communication interface 208 represents 

numerous devices and technologies that allow mobile 
device 200 to send and receive information. The 
devices include wired and wireless modems, satellite 
receivers and broadcast tuners to name a few. Mobile 

25 device 2 00 can also be directly connected to a 
computer to exchange data therewith. In such cases, 
communication interface 208 can be an infrared 
transceiver or a serial or parallel communication 
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connection, all of which are capable of transmitting 
streaming information . 

Input /output components 2 06 include a 
variety of input devices that have previously been 
found on mobile devices such as a touch- sensitive 
screen, buttons, rollers, and a microphone as well as 
a variety of output devices including an audio 
generator, a vibrating device, and a display. The 
devices listed above are by way of example and need 
not all be present on mobile device 200. 

Mobile device 200 also, includes additional 
input devices under the present invention. Under one 
embodiment, these input devices are connected to the 
mobile device through a separate serial port 250 and 
a peripheral interface controller (PIC) 

microprocessor 252. In other embodiments, these 
additional devices are connected to processor 202 
through communication interface 2 08 and PIC 
microprocessor 252 or through PIC microprocessor 252 
directly. Under one embodiment, a microchip 16C73A 
peripheral interface controller is used as the PIC 
microprocessor. In still further embodiments, PIC 
microprocessor 252 is not present and the input 
devices are connected to processor 2 02 through 
various ports such as serial port 2 50 or through 
communication interface 208. 

Under the embodiment of FIG. 1, The 
additional input devices include two touch sensors 
254 and 256, a forward/back tilt sensor 258, a 
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left/right tilt sensor 260, a proximity sensor 262 
consisting of an infrared transmitter 264 and an 
infrared receiver 266, a digital compass 284, and a 
gravity switch 282. The sensing signals from the 
5 infrared receiver 266, left/right tilt sensor 260, 
forward/back tilt sensor 258, digital compass 284, 
and gravity switch 2 82 are provided through 
respective amplifiers 270, 272, 274, 285 and 287 to 
analog inputs of PIC microprocessor 252. These 
10 analog inputs are connected to analog- to-digital 
H converters within PIC microprocessor 2 52. In other 

2? embodiments, the sensors provide a digital output and 

SI thus are connected to digital inputs on the 

%Z microprocessor. 

15 In the embodiment of FIG. 1, touch sensors 

Ss 

3 2 54 and 2 56 are provided to a separate peripheral 

5? interface controller microprocessor 276 which 

□ converts the touch signals into digital values and 

m ■ 

provides the digital values to PIC microprocessor 
^ 20 252. In other embodiments, touch sensors 254 and 256 

are connected directly to analog or digital inputs in 
PIC microprocessor 252 instead of being connected to 
PIC 276 or are connected to processor 202. 

PIC microprocessor 252 also includes a 
25 connection to the ^ power bus of mobile device 200, 
which is shown as connection 278 in FIG. 1. PIC 
microprocessor 252 also includes a connection to a 
power switch 280, which enables PIC microprocessor 
252 to turn mobile device 200 on and off. Note that 



PIC microprocessor 252 always receives power and, 
under one embodiment, is able to control which of the 
sensors receives power at any one time. This allows 
PIC microprocessor 2 52 to manage power consumption by 
5 only sending power to those sensors that it 
anticipates will need to be active. 

Under one embodiment, PIC microprocessor 
252 continuously samples the sensors and transmits 
packets representing the state of these sensors at a 

10 rate of approximately 4 00 samples per second through 
serial port 250. In some embodiments, samples are 
reported at lower speeds to conserve power and 
processing resources . Some sensors may be reported 
at different sampling rates than others (e.g. tilt 

15 may be updated more frequently than touch) . 

Under one embodiment, touch sensor 2 54 is a 
capacitive touch sensor that is divided into two 
regions. In other embodiments, it is also possible to 
implement this sensor with a single detector pad. 

20 This touch sensor is spread across the back and sides 
of mobile device 200. This is shown in more detail 
in FIGS. 4-6 which show a back, left side view and 
right side view of the outside of mobile device 200. 
In FIGS. 4, 5, and 6, touch sensor 254 is shown as 

25 two regions 300 and 302. Region 300 extends from the 
left side to the back of mobile device 200 and region 
3 02 extends from the right side to the back of mobile 
device 200. When a user touches either section 300 
or 3 02, the capacitance associated with the touched 



section changes indicating that the user has touched 
the device. Note that although the touch sensors are 
shown on the exterior of the device in the embodiment 
of FIGS. 4-6, in other embodiments, the touch sensor 
5 is located beneath an outer covering of the device. 

Touch sensor 2 56 is shown in FIG. 3, which 

r 

is a front view of mobile device 2 00. In the 
embodiment of FIG. 3, touch sensor 256 is located on 
the left bezel of display screen 304. In other 

10 embodiments, touch sensor 256 is located on the outer 
casing on the front portion of mobile device 200, but 
not necessarily on bezel 306 of mobile device 200. 

In some embodiments, the touch sensors 
described above are realized using a plurality of 

15 independent touch sensors that each provides a 
separate touch signal. In other embodiments, the 
touch sensors are replaced with position sensors that 
indicate the location where the user is touching the 
device. Those skilled in the art will recognize that 

2 0 additional touch sensors may be added to the mobile 
device within the scope of the present invention. 

Left/right tilt sensor 260 and forward/back 
tilt sensor 258 are shown as a single dotted element 
308 in FIG. 3. These tilt sensors are embedded 

25 within the casing of mobile device 200 and in one 
embodiment are located at a point about which users 
typically pivot mobile device 200 when tilting the 
device. Note that the tilt sensor's position within 
the mobile device is unimportant as it senses only 
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the angle of its physical attitude with respect to 
gravity. The sensor's angular position within the 
device is important. 

Under one embodiment, an Analog Devices 
5 ADXL05 two-axis linear accelerometer is used for tilt 
sensors 258 and 260. Such a sensor detects 

forward/backward tilt, shown by arrows 310 of FIG. 5, 
and left /right tilt, shown in the bottom view of FIG. 
2 . as arrows 312. The sensor also responds to linear 
10 accelerations, such as those resulting from shaking 
the device. Typically, the tilt sensor has a 
response curve both in the ' forward/back direction and 
left/right direction with the form: 

(T — T \ 

Angle = sin" 1 - EQ. 1 

\ k ) 

15 where T is the tilt sensor value, T c is the sensor 
value at 0° tilt, and k is a gain parameter. In 
embodiments where the sensor cannot detect the sign 
of the gravity vector, it is unable to determine if 
the user is holding the device with the display 

2 0 facing up or down. Gravity switch 2 82 of FIG. 1 is 
thus provided in some embodiments to indicate whether 
the display is facing the ground. In other 

embodiments, a three-axis accelerometer is used to 
provide the sign of the gravity vector. 

25 In addition, the tilt sensors do not 

respond to rotation about an axis running from the 
front to the back of the mobile device. Thus, the 
tilt sensors are unable to sense the spinning of the 
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mobile device on its back when laid on a flat table. 
Digital magnetic compass 284 of FIG. 1 is thus 
provided in some embodiments to indicate this type of 
rotation. In other embodiments, solid state gyros 
5 are used instead of the compass. 

When present, gravity switch 2 82 and 
digital compass 284 are also internal to mobile 
device 200. They are not shown in FIG. 3 to reduce 
the complexity of FIG. 3. 
10 Note that the additional input devices of 

p FIG. 1 do not all have to be present under the 

^ present invention. Different embodiments of the 

H invention will use different numbers of and different 

combinations of these additional sensors. Further, 
y„ 15 additional sensors may be added without affecting the 

= functions of the sensors discussed in the present 

S application. 

D Transmitter 264 and receiver 266 of 

Li % 

p proximity sensor 2 62 are shown in FIG. 3. In the 

^ 2 0 embodiment of FIG. 3, transmitter 2 64 is shown below 

and to the right of receiver 2 66, and both the 
transmitter and receiver are located at the top front 
of mobile device 200. 

Under one embodiment, a timer 265 drives 
25 transmitter 264 at 40 kilohertz and transmitter 264 
is an infrared light emitting diode with a 60° beam 
angle. Under such embodiments, receiver 266 is also 
an infrared receiver that is capable of operating at 
the same frequency as transmitter 264. The light 
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produced by transmitter 264 bounces off objects that 
are near mobile device 2 00 and the reflected light is 
received by receiver 266. Receiver 266 typically has 
an automatic gain control such that the strength of 
5 the received signal is proportional to the distance 
to the object. 



embodiment of the proximity sensor. In FIG. 7, the 
sensor value is shown along horizontal axis 702 and 

10 the actual distance to the object is shown along 
vertical axis 704. The graph of FIG. 7 is divided 
into three ranges. Range 706 extends from a distance 
of approximately 2 7 centimeters to infinity and 
indicates that no objects are within range of mobile 

15 device 200. Range 708 extends from approximately 7 
centimeters to 2 7 centimeters and indicates that at 
least one object is within range of mobile device 
200. Readings in third range 710, which extends from 
7 centimeters to 0 centimeters, are considered to be 

20 close to mobile device 200. The response curve of 
FIG. 7 is described by the following equation: 



FIG. 7 shows a response curve for one 




k 



EQ. 2 



25 where z„ m is the distance in centimeters to the 

cm 

object, p is the raw proximity reading, is the 

maximum sensor reading, c is a constant, a is a 
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nonlinear parameter (.77 in one embodiment), and k is 
a gain factor. 

Under one embodiment, the power consumed by 
proximity sensor 262 is limited by pulsing 
transmitter 264 a few times a second when the user is 
out of range, or by reducing the duty cycle of timer 
265. 

In other embodiments, IR receiver 2 66 
generates a digital signal instead of the analog 
signal shown in FIG. 1. The digital signal provides 
a representation of the transmitted signal. However, 
as the distance between the device and the user 
increases, the number of errors in the digital signal 
increases. By counting these errors, PIC 252 is able 
to determine the distance between the user and the 
device. 

FIG. 8 provides a block diagram of the 
software components of one embodiment of the present 
invention. In FIG. 8, a context information server 
800 receives the sensor data from serial port 250 of 
FIG. 1. 

Context information server 800 acts as a 
broker between the sensor values received by the 
microprocessor 252 and a set of applications 802 
operating on mobile device 200. Context information 
server 800 continuously receives sensor data packets 
from PIC 252, converts the raw data into a logical 
form, and derives additional information from the 
sensor data. 
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Applications 802 can access the logical 
form of information generated by context information 
server 800 either by polling a block of shared memory 
804 in which context information server 800 stores 
5 the logical form information, or alternatively by 
asking context information server 800 to provide a 
specific piece of information via a system event 
message when a specific sensor value changes. 

In addition, applications can post certain 
10 context information with context information server 
□ 800 so that it may be shared with other applications. 

J Such posting is described in greater detail in 

context with one embodiment of the present invention 
Jrl described below. 

.Jl 15 In FIG. 8, a first in/first out memory 

^ stack 806 is also provided that stores a history of 

m. past states for mobile device 200. These past states 

y are used in certain embodiments of the present 

O invention as described further below. 

^ 20 Tables 1, 2, 3 and 4 below provide lists of 

the context variables that can be generated by 
context information server 800. In the description 
column of each table, specific values for the 
variables are shown in italics. For example, the 
25 D I S PLAYOR I ENTAT I ON variable can have values of flat, 
portrait, landscape left, landscape right, or 
portrait upside down. 
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Group 


Context Variable 


Description 


Touch 


Holding &Durat ion 


Whether or not 
the user is 
holding the 
device and for 
how long 


TouchingBezelSc 
Duration 


Whether user is 
touching screen 
bezel and for 
how long 



Table 1 



• 

-16- 





Group 


Context Variable 


Description 






TiltAngleLR, 
TiltAngleFB 


Left/Right and 
Forward/Back 
tilt angles in 
degrees 






DisplayOrientation 


Flat, Portrait, 






, Refresh 


LandscapeLe f t , 
LandscapeRigh t , 
or Portrait- 
UpsideDown. A 
Refresh event is 
posted if apps 
need to update 








orientation 


i.T"i 

LU 


Tilt/ 
Acceleroinefcer 


HzLR, MagnitudeLR, 
HzFB , Magnitude FB 


Dominant 
frequency and 
magnitude from 
FFT of tilt 


==3 






angles over the 


%j| 






last few seconds 






LookingAt , 


If user is 






Duration 


looking at 
display 






Moving & Duration 


If device is 








moving in any 








way. 






Shaking 


If device is 
being shaken 






Walking, 


If user is 






Duration 


walking 



Table 2 
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Group 


Context Variable 


Description 


Proximity 

c 


Proximity 


Estimated 
distance in cm 
to proximal 
object 


ProximityState , 
Duration 


Close, InRange, 
OutOf Range, 
Ambi en tLigh t 
(when out-of- 
range and bright 
ambient light is 
present) 


Table 3 


Group 


Context Variable 


Description 


Other 


Scrolling 


If user is 
currently 
scrolling 
(posted by 
scroll app) 


VoiceMemoGesture 


If recording a 
voice memo, 
(posted by voice 
recording app) 



Table 4 



The context variables of Table 1 are 
generated based on signals from the touch sensors, 
those in Table 2 are generated based on tilt sensors, 
those in Table 3 are generated based on the proximity 
sensors, and those in Table 4 are posted by other 
applications and are not generated directly from the 
sensor data. 
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The sensors described in FIG. 1 have many 
novel uses under the present invention. Each of 
these uses is described below. 

ACTIVATING AN AUDIO RECEIVER 
5 Many applications on mobile devices 

activate an audio receiver to receive an audio input 
that is further processed. For example, the audio 
signal can be recorded, can be used as input to a 
speech recognition system, or can be transmitted. 

10 Typically, devices have activated their 

audio receiver based on the user pressing a button or 
activating a particular application. Thus, the user 
must physically manipulate a button or keyboard in 
order to activate and deactivate the audio receiver. 

15 The present invention provides a means for 

automatically activating an audio receiver based on 
how the user is holding the * input device. 

In its most basic form, the present 
invention activates the audio receiver based on a 

2 0 proximity sensor that senses when the input device is 
close to the user. In more complex embodiments, the 
input device must also be held by the user before the 
audio receiver will be activated; this prevents 
accidental activation of the audio receiver when the 

25 device is in a purse or briefcase. In the embodiment 
of FIGS. 2-6, the user is considered to be holding 
the device when the touch sensors on the side and 
back of the device are activated. 
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In still more complex embodiments, the user 
must also tilt the device before the audio receiver 
will be activated. For instance, in one embodiment, 
the user must hold mobile device 2 00 in an almost 
5 vertical position. Under one embodiment, the angle of 
tilt must be within a specific range. Since users 
typically tilt an input device to the left and back 
when making the gesture associated with wishing to 
speak into the input device, both left /right tilt and 

10 . forward/back tilt can be examined to determine if the 
tilt is proper. These tilt angles correspond to the 
approximate angle at which one would hold a device 
such as a mobile phone when speaking into it. 
Several ranges of tilt signals can be used, such that 

15 tilting will be recognized if the device is held 
either in the left or the right hand. 

This can be seen in FIG. 9, which shows 
left/right tilt along horizontal axis 1000 and 
forward/back tilt along vertical axis 1002. A block 

20 1004 indicates the acceptable ranges for left/right 
and forward/back tilt for activating the audio 
receiver. 

The conditions of being held, having the 
device in close enough proximity, and having the 
25 device properly tilted must all be met for at least 
.1 seconds before the audio receiver is activated 
under one embodiment of the present invention. 

In embodiments that use proximity, touch, 
and tilt to determine when to activate the audio 
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receiver, the audio receiver is deactivated if any- 
one of these criteria stops being met. For example, 
if the user stops holding the device, or the user 
takes the device away from their mouth, the audio 
5 receiver would be deactivated. 

Under some embodiments, to assist the user 
in knowing when the audio receiver is active, a sound 
is emitted by mobile device 200 when the receiver is 
activated. Another sound is then emitted when the 
10 audio receiver is deactivated. In still further 
embodiments, a separate sound is provided to indicate 
that the gesture is proper for activating the audio 
receiver before the audio receiver is actually 
activated . 

15 Using the context variables of Tables 1-4, 

an application would determine whether to activate 
its audio receiver based on the HOLDING and DURATION 
context variable, the TiltAngleLR and TiltAngleFB 
context variables, and the Proximity or 

2 0 ProximityState context variables. In some 

embodiments, the sequence of recent values is used to 
determine a composite gesture, rather than just 
comparing instantaneous values . 

As noted above, the activation of an audio 

25 receiver can be used when recording an audio message, 
when applying an audio signal to a speech recognition 
system or when transmitting a voice signal either 
across a computer network or through a cellular 
network. In addition, the activation of an audio 
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receiver can be made through a phone, or any hand 
held device that is capable of receiving audio 
signals . 

It is thought by the present inventors that 
5 by eliminating the need to press an activation 
button, the user is able to concentrate more on other 
tasks while speaking into the audio receiver. Thus, 
since the user does not have to concentrate on 
pressing and maintaining tension on a particular 

10 button, they are better able to concentrate on other 
tasks while speaking. 

Although the above embodiment has been 
described in connection with activating an audio 
receiver, it may be generalized to starting other 

15 applications based on the mobile device being placed 
in an appropriate orientation for the application. 
For example, if the user is holding the device in an 
orientation that is indicative of listening to the 
device, an application that automatically plays voice 

2 0 mail could be activated. Another example would be 
answering a call on a cell phone: when the phone 
rings, the user picks it up and places it next to his 
or her ear to answer the call. Because of the sensed 
gesture, there is no need to perform secondary 

25 preparatory actions to answer the call (such as 
pressing a TALK button or opening a flip- top cover) . 
Such preparatory actions are unnecessary under the 
present invention and as such the present invention 
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eliminates extra steps that may be distracting or 
unfamiliar to the user. 

CHANGING DISPLAY ORIENTATION 
In other embodiments of the present 
5 invention, the tilt sensor is used to detect the 
orientation of the mobile device so that the image on 
the display of the mobile device may be matched to 
the mobile device orientation. 

FIG. 10 provides an example of a mobile 
10 device 1100 in an upright orientation. In FIG. 10, 
PI the present invention displays an image 1102 of a set 

^ of text in a portrait orientation to match the 

SJ orientation of mobile device 1100. FIG. 11 shows the 

^ same mobile device rotated counterclockwise 90°. 

Nl 15 Under the present invention, this rotation is sensed 

_ " by the tilt sensors and in response, a new image 1104 

!f of the set of text is displayed. In particular, 

o image 1104 shows the text in a landscape view to 

^ match the new orientation of mobile device 1100. 

H» 2 0 Under the present invention, the displayed 

image may be matched to the orientation of the device 
through a simple rotation of the display image or 
through a redrawing of the display image based on the 
orientation of the device and the relative dimensions 
25 of the display in that orientation. 

A simple rotation is typically performed by 
using the center of the screen as the center of 
rotation. For such rotations, a transform of points 
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x,y in the document to a point x' ,y' on the screen is 
given by equation 3: 



m = T R-T~ ] EQ. 3 

5 

where T is a translation and R is the 2D rotation 
matrix for 0°, 90°, 180° , or 270°. 

A redrawing of the image that takes the 
dimensions of the display into consideration allows 

10 applications to change the contents of the image 
before redrawing the image. For instance, a word 
processing application can recalculate word wrapping 
based on the dimensions of the display in its new 
orientation. Thus, the number of words per line will 

15 change when the image is drawn in its new 
orientation. An example of this effect can be seen 
by comparing the text in image 1102 to the text in 
image 1104. 

Context information server 800 causes the 
2 0 image to be refreshed by sending an appropriate 
notification to applications 802 . These applications 
then generate a new image for the display. 

To determine the orientation of the mobile 
device, most embodiments of the present invention 
25 examine both the left/right tilt of the mobile device 
and the front/back tilt. FIG. 12 provides a graph 
showing the orientations determined by context 
information server 800 for various combinations of 
left/right and forward/back tilt. In FIG. 12, 
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left/right tilt is shown along horizontal axis 1300 
and forward/back tilt is shown along vertical axis 
1302. In FIG. 12, there are four orientation regions 
1304, 1306, 1308, and 1310, which are separated by 
5 deadbands 1312, 1313, 1314 and 1315. FIG. 12 also 
includes a flat area 1316, that corresponds to the 
mobile device being laid flat. 

Orientation area 13 : 04 is related to an 
upright portrait orientation for a mobile device such 

10 as mobile device 1100 of FIG. 10. This is the typical 
or natural way for a user to hold the mobile device. 
Orientation areas 1308 and 1310 are associated with a 
rotation of the mobile device counterclockwise and 
clockwise 90° , respectively. Orientation area 1306 

15 is associated with the mobile device being rotated 
180° so that is upside down. 

Transitions between these areas nominally 
occur when the left/right tilt equals the 
forward/back tilt. As shown by deadbands 1312, 1313, 

20 1314 anci 1315, the display does not switch 
immediately at these angles. Instead, there is a 
plus or minus 5° deadband to prevent jitter. Thus, 
the device must tilt through the entire deadband zone 
before context information server 800 of FIG. 8 will 

2 5 issue a new orientation value. In some embodiments, 
there is a further 0.5 second time-out. The 
orientation of the device must remain at the new 
orientation for 0.5 full seconds before the display 
format changes . 
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Under some embodiments of the present 
■ invention, the mobile device ! s directional inputs are 
also remapped to match the orientation of the mobile 
device. For example, directional input 1106 of FIGS. 
5 10 and 11 has a different mapping in the two figures. 
In FIG. 10, pressing directional input 1106 toward 
the screen corresponds to moving upward through the 
text. In FIG. 11, pressing directional input 1106 
toward the screen corresponds to moving left through 
10 the text. 

Under one embodiment of the present 
invention, the orientation of the display is 
stabilized to avoid undesired switching of the 
displayed orientation as the user places the mobile 

15 device on a flat surface. During the process of 
placing a mobile device on a flat surface, it is 
common for the user to tip the mobile device slightly 
in one or more directions. Without the stability 
provided by this embodiment, the display may switch 

20 to other orientations that the user does not want. 

Under one embodiment of the present 
invention, orientation stability is maintained by 
using a first-in-first-out queue of recent display 
orientations. Such a queue is shown as FIFO 806 in 

25 FIG. 8. The FIFO-queue includes a list of recent 
display orientations. 

When the user puts down the device, 
indicated by the tilt sensors 'indicating the device 
.is flat and the touch sensors indicating that the 
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device is not being held, context information server 
800 searches through FIFO 806 to find the most recent 
stable orientation other than flat. An orientation 
is considered stable if it was maintained for more 
5 than 1 or 2 seconds. This stable orientation is then 
selected as the orientation to be used while the 
device remains flat. 

TILT SCROLLING 
Under embodiments of the present invention, 

10 a user can scroll through a displayed document, 
spreadsheet, or image, simply by tilting the input 
device. In most embodiments, the user indicates 
their desire to use tilt scrolling by touching the 
touch pad on the bezel of the mobile device. 

15 Although in the embodiments discussed below, a touch 
sensor is used to indicate a desire to begin tilt 
scrolling, other inputs can be used within the scope 
of the present invention to indicate that the user 
wishes to begin tilt scrolling. 

2 0 Under one embodiment of the invention, when 

the user indicates that they wish to begin tilt 
scrolling, the mobile device captures the current 
forward/back tilt and left/right tilt and uses these 
tilt angles as the starting orientation for the 

25 device. The direction and speed of scrolling is then 
determined relative to this starting orientation. 

By capturing this starting orientation, the 
present invention keeps the displayed image from 
scrolling until the user tilts the device from its 
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initial position. This makes the scrolling behave in 
a more predictable manner and is different from prior 
art tilt scrolling devices. In prior art devices, 
the direction and speed of the scrolling is 
5 determined relative to tilt angles, of zero or tilt 
angles that the user must set using a separate 
command to "recenter" the device. As such, the 
device will begin to scroll immediately if the device 
is tilted when the user indicates that they want to 

10 activate tilt scrolling. The present invention 
improves on the use of a "recenter" command by 
combining recentering with the initiation of 
scrolling (i.e. as soon as the user touches the bezel 
to start scrolling, this also resets the starting 

15 angles for the tilt) . 

Under the present invention, the rate and 
direction of forward/back scrolling is related to the 
angle of forward/back tilting by: 

v /b =k- sgn(^ ) ■ max^ || - dA^ ,0)« EQ . 4 

2 0 where v fb is the calculated velocity for the forward- 
backward scrolling, k is the control gain, dA fb is the 
change in the forward/back tilt angle relative to a 
beginning orientation, dA m i n is the size of a dead 
band, and a is a non- linear parameter. Under one 

2 5 embodiment, k= . 2 , a =1.6 and dA min =l°-4°. A similar 
equation is used to determine left/right scrolling 
based on left/right tilt angle. 

The scrolling control can be single axis 
where the device examines the left/right tilt and the 
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f orward/back tilt and only scrolls along the axis 
with the larger tilt angle. Thus, if the relative 
forward/back tilt is larger than the relative 
left /right tilt, the image is scrolled based only on 
5 the forward/back tilt. Alternatively, the scrolling 
control may be mixed exclusive axis, where a square 
dead band is applied around the zero tilt point, such 
that small tilts are ignored- For instance, if a 
user tilts the device a small amount forward but a 

10 large amount left, the image will only scroll left. 
However, if the user increases the forward tilt, the 
image will scroll diagonally. Lastly, the scrolling 
control can be a dual axis control in which both 
forward/back tilt and left/right tilt are used at the 

15 same time to scroll the image without a dead band. 

Under one embodiment of the present 
invention, tilt scrolling is combined with the 
orientation matching described above. This 
represents a significant challenge since both 

20 techniques utilize the tilting of the device for 
changing the display. 

Under the present invention, once scrolling 
has been initiated, the tilt of the device is only 
applied to the scrolling and is not used to change 

25 the orientation of the display. Thus, tilting the 
mobile device will not cause the orientation to 
• switch until scrolling is deactivated. 

The application that is controlling the 
scrolling helps to enforce this rule by posting a 
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scrolling value with Context Information Server 800. 
This value is shown in Table 4. Once this value is 
1 posted, Context Information Server 800 suspends 

updates to the D I S PLAYORI ENTAT I ON value until 
5 scrolling v has stopped. 

In addition, when the user releases the 
scrolling button or the scrolling touch sensor, the 
system does not automatically change the orientation 
based on the current tilt. Instead, the system 
10 allows the user 1 or 2 seconds to return the mobile 
y device to the position it was in when scrolling was 

ED initiated. After that waiting period, the orientation 

of the image is changed to match the orientation of 
=P the mobile device. 

%j 15 SCROLLING WITHOUT MENU BAR AND START BAR 

L, In many applications, a displayed document 

01 is shown with banners that provide menus, scroll 

5j bars, and/or command icons. Under one embodiment of 

the present invention, the inventors have discovered 
2 0 that users do not need the command menus or the 
command icons during scrolling. Thus, under one 
embodiment of the invention, applications remove 
their menu bars and command bars from the display 
during scrolling. Note that this invention is not 
2 5 limited to mobile devices and may be used on any 
computing system or with any type of mobile device. 
By removing the menu bars or command icons, more 
screen area is available for showing the image that 
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is being scrolled- This makes it easier for the user 
to find the information they want. 

The removal of the command bars during 
scrolling is independent of tilt scrolling. In other 
5 words, the aspect of removing command bars may be 
practiced- regardless of how the scrolling is 
initiated . 

Under one embodiment , the input used to 
remove the menu bars and command icons is the same as 

10 that used to initiate scrolling. For example, in a 
tilt scrolling application, the menu bar and command 
icons * disappear as soon as the user presses the 
button or touch sensor that indicates that tilt 
scrolling is to be initiated. Under such 

15 embodiments, a larger dead zone from the zero 
orientation angle may be needed to ensure that the 
image does not scroll when the user just wants to 
view the full screen. 

Selective Display of Toolbars 

2 0 Under some embodiments of the present 

invention, several touch sensors are provided on the 
mobile device. For a given application, each touch 
sensor is associated with a different toolbar. When 
the user is touching a particular touch sensor, its 

25 corresponding toolbar is displayed. However, when 
the user is . not touching the touch sensor, the 
toolbar is hidden so that more of the document or 
image may be displayed. In some embodiments, the 
touch sensor may be associated with an entire 
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application (rather than just a toolbar) so that the 
user may switch applications temporarily by 
maintaining contact with the touch sensor. In this 
case, further interaction, with the application using 
the stylus (touch-screen) causes the switch to become 
permanent . 

Display Contrast Adjustment 
One problem encountered when using a tilted 
liquid crystal display is that as the display is 
tilted away from the user, the contrast of the 
displayed image is lessened. The present invention 
improves the displayed image by increasing the 
contrast of the display as the user tilts the display 
away from them. Note that this aspect of the present 
invention is not limited to personal information 
managers and may be used with any LCD device such as 
a pager, watch, laptop computer, cell phone, or stand 
alone LCD display. Under one embodiment, the 

contrast is set according to the following equation: 

contrast = m • dA^ +b EQ . 5 

where m is a constant equal to -.13 5, b is a constant 
equal to 5.25, dA fb is the change from the ideal 
viewing angle as measured by a tilt sensor in the 
display, and "contrast" is the change in the contrast 
setting. Note that these values correspond to the 
software contrast settings of 1-10 available on the 
Cassiopeia E105 device. To prevent hysteresis, a tilt 
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angle change of more than 3° is typically required 
before the contrast is adjusted. 

POWER MANAGEMENT 
Under one embodiment of the present 
5 invention, a mobile device places itself in a full- 
power mode based on how the device is being handled. 
In one particular embodiment, the invention uses a 
combination of sensors to determine whether the user 
wants the device to be in full -power mode. In 

10 particular, the system uses a touch sensor to 
determine whether the user is holding the device and 
a tilt sensor to determine whether the user has 
properly oriented the device so that it is likely 
they are looking at it. By using both a touch sensor 

15 and an orientation sensor, the present invention 
avoids placing the device in full -power mode when it 
is in a briefcase. 

In one particular embodiment, the 
orientation for full-power mode is a left/right tilt 

20 of between plus or minus 15°, a forward/back tilt of 
greater than -5°, and an upright portrait 
orientation. If the user maintains this orientation 
for at least .5 seconds, PIC microprocessor 252 of 
FIG. 1 places the device in full -power mode using 

25 power switch 280. The required time interval can be 
set as desired but is used to prevent powering up due 
to transient signals. 

In other embodiments, the present invention 
prevents a mobile device from entering an idle mode 
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if the user is handling the device or gesturing 
toward the device. In idle mode, the mobile device 
reduces the power consumption of the device by 
turning off the display. Typically, a mobile device 
5 will enter idle mode if the user has not pressed a 
button or touched the screen for some period of time. 
Under certain types of usage, users find that the 
system powers down when they would rather have it 
remain active. For instance, this often happens when 

10 the user is reading a large amount of text or is 
trying to interact with someone else while 
periodically relating to the text. 

Under one embodiment of the invention, the 
device is prevented from entering an idle mode when 

15 context information server 800 determines that the 
user is holding the input device or when it 
determines that there is motion near the device. Such 
motion can be detected by changes in the output of 
the proximity sensor and is indicative of a user 

2 0 gesturing toward the device. 

Note that this embodiment relies on motion 
and not just proximity. This is done to allow the 
device to enter idle mode when it is placed near a 
fixed object while it is not being used. For example, 

25 under the present invention, the device will still 
enter idle mode if a stack of paper is placed on top 
of the device. 

Although the present invention has been 
described with reference to preferred embodiments, 
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workers skilled in the art will recognize that 
changes may be made in form and detail without 
departing from the spirit and scope of the invention. 



